Tech Setup (Instance ID: i-0026e1085400d6372)

I set up my EC2 instance with the following configurations:

Stream Processing Application

The instance’s setup was successful, logging in to RStudio with username ceu and password ceudata:

I set up an app.properties file in the streamer folder, set the stream name to CEU-Binance, the region to Ireland (eu-west-1) and named my application patrik-szigeti-assignment-app:

I created app.R based on the file from the lecture with the following modifications:

app.log logs all incoming batches with additional information about the received data:

Jenkins

I created a new Jenkins job as a Freestyle project under the name CEU-Binance Assignment that I later renamed to ceu-binance-assignment to keep consistency with the naming convention I applied so far.

The script being called from Jenkins is the overall_value.R file which connects to Redis and creates a list from the key-value pairs that are stored in the Redis cache. As these items are coming through as a list, I have to unnest the values with lapply, and then cast them as.numeric. After dropping the original list, I’m able to aggregate the column and calculate the sum of the transactions in the last batch. The overall_transaction_value function takes care of the data processing and transformation, and finally prints out the overall value of the transaction in USD.

Unfortunately I kept running into a “file not found” issue in Jenkins when trying to run Rscript /home/overall_value.R, so I ended up pasting the whole code snippet into “Execute shell command”:

The job ran successfully (after a couple of tries, this was run #20), and returned the overall value for the last batch:

Plotting and Slack

I created two plots to post to #bots-final-project, but before that, I had to create a data table from the key-value pairs, and then transform the amounts that were coming through as a list to numeric values. I also cleaned up the symbols column a bit, and removed the amount: prefix that I used to store the values in Redis.

I posted both charts to the Slack channel:

Stop the instance

And I didn’t forget to stop my instance either: